Categorizing, classifying, and identifying network flows using network and host components

ABSTRACT

Network flows are identified by analyzing network traffic and network host information. The network host information may be collected by network host monitors associated with network hosts. Network traffic and network host information are evaluated against network flow profiles to identify network flows. If a network flows are identified with high certainty and are associated with previously identified network applications, then network flow policies can be applied to the network flows to block, throttle, accelerate, enhance, or transform the network flows. If a network flow is identified with lesser certainty or is not associated with a previously identified network application, then a new network flow profile can be created from further analysis of network traffic information, network host information, and possibly additional network host information collected to enhance the analysis. New network flow profiles can be communicated with a service provider for analysis and potential distribution to other networks.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application is a continuation of application Ser. No. 10/610,337,filed Jun. 30, 2003, now U.S. Pat. No. 7,422,852, which claims thebenefit of U.S. Provisional Application No. 60/645,283, filed Jan. 19,2005, which are incorporated herein by reference in their entirety forall purposes.

STATEMENT AS TO RIGHTS TO INVENTIONS MADE UNDER FEDERALLY SPONSOREDRESEARCH OR DEVELOPMENT

NOT APPLICABLE

REFERENCE TO A “SEQUENCE LISTING,” A TABLE, OR A COMPUTER PROGRAMLISTING APPENDIX SUBMITTED ON A COMPACT DISK

NOT APPLICABLE

BACKGROUND OF THE INVENTION

This application relates to the field of computer networks, andspecifically to software and hardware for monitoring and controllingnetwork traffic. Computer networks often include hundreds or thousandsof network hosts. A network host is a computer or other hardware devicethat runs software applications and originates and/or receives networkflows.

Network administrators are often responsible for maintaining thesenetwork hosts in proper running order and to ensure the network operatessecurely and reliably. To that end, network administrators often setrules or network policies about the types of software applications andnetwork traffic allowed on a network.

Network applications are software applications on a network host thatare responsible for originating and/or receiving network traffic flows,referred to as network flows. Some network applications are well-behavedand conform with a network's rules and policies. Other networkapplications are poorly-behaved, installing without the user's ornetwork administrator's permission, hiding themselves and theiroperation, and violating a network's rules and policies. Examples ofpoorly-behaved network applications include computer viruses, worms, andspyware and malware applications. Additionally, some more legitimateapplications, such as instant messaging applications, file-sharing orother types of peer-to-peer network applications, voice-over IP (VOIP)communication applications, and multimedia applications are responsiblefor network flows that can circumvent network policies and jeopardizenetwork security and reliability.

Often, poorly-behaved network applications will attempt to conceal theirnetwork flows to avoid detection and disregard network policies. Commonevasion techniques include using non-standard network protocols, dynamicport and channel selection, which limits the effectiveness of monitoringand blocking network ports to control network traffic; HTTP/HTTPStunneling, which hides network flows in normally-permitted web traffic;Peer-to-Peer onion routing, which selects destination addresses forpeer-to-peer routing at random to circumvent destination addressblocking; and encryption of network packet data, which prevents networkmonitors from examining the contents of network packets to identify thetype of network flow.

For example, some common peer-to-peer VOIP applications circumventnetwork policies in a number of ways. The peer-to-peer VOIP applicationcan dynamically selected different ports and channels for communication.If UDP is blocked, the application can fall back on TCP/IP.Additionally, the peer-to-peer VOIP application can tunnel its data overopen ports 80 or 443, which are normally intended for HTTP or SSLtraffic. A peer-to-peer VOIP application can dynamically selectsupernodes in its peer-to-peer network to circumvent destination addressdetection and blocking. Additionally, data can be encrypted to preventdetection using packet inspection.

Prior network monitoring applications could monitor the content, size,and source and destination addresses of network flows as they passthrough a gateway or other point in the network. However, due to theseevasion techniques, prior network monitoring applications often have toolittle information to reliably identify unauthorized network flows anddetect poorly-behaved network applications.

It is therefore desirable for a system and method to provide improvedcapabilities in categorizing and controlling network flows and networkapplications. It is further desirable for the system and method todetect and adapt to new or revised network applications and networkflows with little or no user intervention.

BRIEF SUMMARY OF THE INVENTION

An embodiment of the invention identifies network flows by analyzingnetwork traffic information and network host information. An embodimentstatistically determines correlations between portions of the networktraffic information and the network host information. In an embodiment,the network host information is collected by network host monitorsassociated with network hosts. Network traffic information and networkhost information is evaluated against network flow profiles to identifynetwork flows. If a network flow is identified with a high degree ofcertainty and is associated with a previously identified networkapplication on a network host, then a network flow policy can be appliedto the network flow to block, throttle, accelerate, enhance, and/ortransform the network flow. If a network flow is identified with alesser degree of certainty or is not associated with a previouslyidentified network application, then an embodiment can create a newnetwork flow profile from further analysis of network trafficinformation, network host information, and possibly additional networkhost information collected to enhance the analysis. New network flowprofiles can be communicated with a service provider for analysis andpotential distribution to other networks.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention will be described with reference to the drawings, inwhich:

FIG. 1 illustrates a system for categorizing and controlling networkflows according to an embodiment of the invention;

FIG. 2 illustrates a method of categorizing and controlling networkflows according to an embodiment of the invention;

FIG. 3 illustrates a method of identifying new or modified network flowsand network applications according to an embodiment of the invention;and

FIG. 4 illustrates a computer system suitable for implementing anembodiment of the invention.

DETAILED DESCRIPTION OF THE INVENTION

FIG. 1 illustrates a system 100 for categorizing and controlling networkflows according to an embodiment of the invention. System 100 includes avariety of components connected with a communications network, which canuse packet-based network protocols such as TCP/IP. System 100 includes acorrelation engine 105. Correlation engine 105 is a software componentadapted to operate on a network gateway, a server computer, or any othertype of computer or other network hardware. Correlation engine 105 isresponsible for identifying the network traffic produced by anapplication, referred to as a network flow, and the identity ofapplications responsible for network flows.

In an embodiment, the correlation engine 105 receives data about networkflows from two different sources. First, a network traffic monitor 120monitors network traffic, or network flows, in system 100. The networktraffic monitor 120 is a software and/or hardware component adapted tooperate on a network gateway, a server computer, or any other type ofcomputer or other network hardware. The network traffic monitor 120collects information on network flows being sent or received by networkapplications within system 100, such as the source and destinationaddresses of network packets, the size of network data in networkpackets, the contents of network packets, the rate of related networkpackets in a network flow, and any other attributes of one or morenetwork packets in a network flow.

Second, the correlation engine 105 receives information from hostmonitor applications. System 100 includes one or more network hosts,such as network hosts 115A, 115B, and 115C. Each network host isassociated with a host monitor application, such as host monitorapplications 110A, 110B, and 110C. In an embodiment, a host monitorapplication can run locally on its associated network host or remotelyon a different network host, server, or other hardware device in thenetwork.

The host monitor application collects information about its associatednetwork host and provides this information to the correlation engine 105for further analysis. Example information capable of being provided bynetwork host monitor applications include the names of executable files,DLLs, and other files installed or stored on a network host; theconfiguration data of a network host, including registry keys;processes, threads, and services executed by a network host; performancedata, such as processor, memory, and disk usage of a network host; thenumber and configuration of open network connects of a network host; thedevice drivers installed or operating in a network host; user input to anetwork host; system hooks connected to an operating system kernel,which may be employed by rootkit applications; and any other attribute,performance parameter, or operating characteristic of a network host.

The correlation engine 105 uses information from the network trafficmonitor 120 and host monitors 110 to reliably identify network flows andassociated network applications. In an embodiment, the correlationengine 105 employs a variety of statistical techniques to identifycorrelations between some or all of the information collected from thenetwork traffic monitor and host monitors 110. These correlations can beused to construct a network flow profile.

A network flow profile enables the identification of a specific networkflow within the entire traffic of a network. Additionally, a networkflow profile can identify the application on a network host responsiblefor originating and/or receiving the network flow. In an embodiment, thecorrelation engine 105 uses Bayesian statistical analysis techniques tocreate and process network flow profiles. The correlation engine 105 canevaluate network flow profiles against the entire network traffic andhost monitor information to get a probability estimate that a givennetwork flow and/or associated network application is present in thenetwork. If the probability estimate is above a given threshold value,then the system can assume that the network flow matches the profile andcan apply the appropriate policy to the network flow.

In another embodiment, if the probability estimate is below a firstthreshold value but above a second threshold value, then the correlationengine 105 can initiate a process to discover additional informationabout the network flow and associated application and create a newnetwork profile. This embodiment is discussed in detail below.

In a further embodiment, the correlation engine 105 can provide newlydiscovered network flow profiles to a service provider 125. As discussedin detail below, the service provider 125 can evaluate new network flowprofiles and choose to disseminate them to correlation engines operatingon other networks.

In an additional embodiment, the correlation engine 105 can interfacewith a policy manager 130. The policy manager 130 enables networkadministrators to set policies for network flows. Policies can be usedto block, throttle, accelerate, enhance, or transform network trafficthat is part of an identified network flow. In an embodiment, policiesfor network flows are enforced by network traffic controlling devicessuch as switches, routers, firewalls, proxies, IPS, and EPS systems. Thecorrelation engine 105 and policy manager 130 can communicate withnetwork traffic controlling devices via any interface or protocol, suchas SNMP.

An embodiment of system 100 can process network flows in three differentways. First, the system 100 can be used to identify network flows andassociated network applications based on correlations of network trafficand host monitor information with one or more network flow profiles.

Second, an embodiment of system 100 can identify new networkapplications responsible for a previously known network flow. Forexample, network applications are often changed or revised.Poorly-behaved network applications, such as virus, worms, and spywarefrequently change characteristics to evade detection and removal. In anembodiment, when a network flow matches a network profile above a firstprobability threshold but the application normally associated with thisnetwork flow is not found on the network host, the correlation engine105 and host monitor 110 can collect additional information to identifythe actual network application associated with the network flow and tocreate a new or revised network flow profile.

Third, an embodiment of system 100 can identify new network flowsassociated with a previously known network application. This can occurwhen a host monitor reliably identifies an application responsible for anetwork flow, but the network flow itself does not match any networkflow profile. In response, the correlation engine 105 collectsadditional information to create a network flow profile. In anembodiment, system 100 attempts to monitor and identify all networkflows of all network applications. In another embodiment, system 100only monitors and identifies all network flows associated with a subsetof the network applications, such as those already designated aspoorly-behaved.

FIG. 2 illustrates a method 200 of categorizing and controlling networkflows according to an embodiment of the invention. At step 205, method200 determines whether a network flow matches a previously creatednetwork flow profile. In an embodiment, this is determined bycalculating a probability that the network flow matches the networkprofile. If so, then method 200 proceeds to step 210. Step 210 accessesinformation from a host monitor to determine if the network flow isassociated with a known network application. If step 210 determines thatthe network flow is produced by the network application normallyresponsible for this type of network flow, then step 215 enforces thepolicy associated with this network flow.

Conversely, if step 210 determines that the known network flow is notassociated with its normal application, then method 200 proceeds to step220. Step 220 builds a new network flow profile for the known networkflow and the newly discovered associated application. The details ofstep 220 and 230 are discussed in detail below.

Returning to step 205, if the network flow does not match a network flowprofile, then method 200 proceeds to step 225. Step 225 uses informationfrom the host monitor to determine if the unknown network flow isassociated with a known network application. If so, then method 200proceeds to step 230 to build a new network flow profile for the newlydiscovered network flow associated with the known network application.

Conversely, if the unknown network flow is not associated with a knownnetwork application, then method 200 proceeds to step 220 to build a newnetwork profile based on the newly discovered network application andnetwork flow.

FIG. 3 illustrates a method 300 of identifying new or modified networkflows and network applications according to an embodiment of theinvention. Method 300 receives information about a detected network flowand/or network application from step 220 or 230 of method 200. Step 305identifies the network host and corresponding host monitor associatedwith the network flow.

Step 310 contacts the identified host monitor and instructs it to gatheradditional information about the network host and its applications. Inan embodiment, host monitors normally only track as much information asnecessary for the current network flow profiles to work correctly. Thisis done to minimize the burden on the network host by the host monitor.However, in an embodiment, the identified host monitor expands itsinformation gathering efforts in response to an instruction from thecorrelation engine to provide the correlation engine with as much dataas possible for analysis of an unknown network flow and/or networkapplication.

Step 315 receives host information from the identified host monitor.Step 315 analyzes the host information as well as information from thenetwork traffic monitor to determine correlations between information.

Step 320 uses these correlations to build a new network flow profile tobe used to identify the network flow and/or network application. Asdiscussed above, statistical analysis and Bayesian probability networkscan be used to create and evaluate network flow profiles.

Following step 320, optional step 325 forwards the new network flowprofile to a service provider for further analysis, refinement, andpossible dissemination to other networks and correlation engines.

FIG. 4 illustrates a computer system 1000 suitable for implementing anembodiment of the invention. Computer system 1000 typically includes amonitor 1100, computer 1200, a keyboard 1300, a user input device 1400,and a network interface 1500. User input device 1400 includes a computermouse, a trackball, a track pad, graphics tablet, touch screen, and/orother wired or wireless input devices that allow a user to create orselect graphics, objects, icons, and/or text appearing on the monitor1100. Embodiments of network interface 1500 typically provides wired orwireless communication with an electronic communications network, suchas a local area network, a wide area network, for example the Internet,and/or virtual networks, for example a virtual private network (VPN).

Computer 1200 typically includes components such as one or more generalpurpose processors 1600, and memory storage devices, such as a randomaccess memory (RAM) 1700, disk drives 1800, and system bus 1900interconnecting the above components. RAM 1700 and disk drive 1800 areexamples of tangible media for storage of data, audio/video files,computer programs, web browsers, applet interpreters or compilers,virtual machines, and embodiments of the herein described invention.Other types of tangible media include floppy disks; removable harddisks; optical storage media such as DVD-ROM, CD-ROM, and bar codes;non-volatile memory devices such as flash memories; read-only-memories(ROMS); battery-backed volatile memories; and networked storage devices.

Further embodiments can be envisioned to one of ordinary skill in theart after reading the attached documents. In other embodiments,combinations or sub-combinations of the above disclosed invention can beadvantageously made. The block diagrams of the architecture and flowcharts are grouped for ease of understanding. However it should beunderstood that combinations of blocks, additions of new blocks,re-arrangement of blocks, and the like are contemplated in alternativeembodiments of the present invention.

The specification and drawings are, accordingly, to be regarded in anillustrative rather than a restrictive sense. It will, however, beevident that various modifications and changes may be made thereuntowithout departing from the broader spirit and scope of the invention asset forth in the claims.

1. A method of processing network traffic, the method comprising:receiving network traffic information, wherein the network trafficinformation describes attributes of network traffic in a network;receiving network host information, wherein the network host informationdescribes attributes of a network host associated with at least aportion of the network traffic; analyzing the network trafficinformation and the network host information to identify at least onenetwork flow within the network traffic; determining if the network flowmatches a network flow profile; and applying a policy to the networkflow in response to a determination that the network flow matches thenetwork flow profile.
 2. The method of claim 1, further comprising:creating a new network flow profile in response to the determinationthat the network flow does not match the network flow profile.
 3. Themethod of claim 2, wherein creating a new network profile includesdetermining a correlation of at least a portion of the network trafficinformation and at least a portion of the network host information withthe network flow.
 4. The method of claim 3, wherein creating a newnetwork profile includes creating a Bayesian probability network basedon the correlation.
 5. The method of claim 2, wherein creating the newnetwork flow profile comprises: requesting additional network hostinformation in response to the determination that the network flow doesnot match the network flow profile; receiving the additional networkhost information; and further analyzing the network traffic information,the network host information, and the additional network hostinformation to create the new network flow profile.
 6. The method ofclaim 1, further comprising: determining if the network flow isassociated with a previously identified application; and creating a newnetwork flow profile in response to the determination that the networkflow at least partially matches the network flow profile and thedetermination that the network flow is not associated with thepreviously identified application.
 7. The method of claim 1, wherein thenetwork host information is collected by at least one network hostmonitor.
 8. The method of claim 1, wherein the network host informationincludes a program installed on the network host.
 9. The method of claim1, wherein the network host information includes a program executed bythe network host.
 10. The method of claim 1, wherein the network hostinformation includes configuration data of the network host.
 11. Themethod of claim 1, wherein the network host information includesperformance data of the network host.
 12. The method of claim 1, whereinthe network host information includes network connection data of thenetwork host.
 13. The method of claim 1, wherein the network hostinformation includes user input to the network host.
 14. The method ofclaim 1, wherein the network host information includes system hooksassociated with an operating system of the network host.
 15. The methodof claim 2, further comprising communicating the new network flowprofile with a service provider.
 16. The method of claim 1, whereinapplying the policy to the network flow includes communicatingidentifying information of the network flow to a network traffic controldevice.
 17. The method of claim 16, wherein the identifying informationof the network flow is communicated with the network traffic controldevice via a network management protocol.
 18. The method of claim 1,wherein the network traffic information is collected by at least onenetwork traffic monitor.
 19. An information storage medium comprising aplurality of instructions adapted to direct an information processingdevice to perform an operation comprising: receiving network trafficinformation, wherein the network traffic information describesattributes of network traffic in a network; receiving network hostinformation, wherein the network host information describes attributesof a network host associated with at least a portion of the networktraffic; analyzing the network traffic information and the network hostinformation to identify at least one network flow within the networktraffic; determining if the network flow matches a network flow profile;and applying a policy to the network flow in response to a determinationthat the network flow matches the network flow profile.
 20. Theinformation storage medium of claim 19, further comprising: creating anew network flow profile in response to the determination that thenetwork flow does not match the network flow profile.
 21. Theinformation storage medium of claim 20, wherein creating a new networkprofile includes determining a correlation of at least a portion of thenetwork traffic information and at least a portion of the network hostinformation with the network flow.
 22. The information storage medium ofclaim 21, wherein creating a new network profile includes creating aBayesian probability network based on the correlation.
 23. Theinformation storage medium of claim 20, wherein creating the new networkflow profile comprises: requesting additional network host informationin response to the determination that the network flow does not matchthe network flow profile; receiving the additional network hostinformation; and further analyzing the network traffic information, thenetwork host information, and the additional network host information tocreate the new network flow profile.
 24. The information storage mediumof claim 19, further comprising: determining if the network flow isassociated with a previously identified application; and creating a newnetwork flow profile in response to the determination that the networkflow at least partially matches the network flow profile and thedetermination that the network flow is not associated with thepreviously identified application.
 25. The information storage medium ofclaim 19, wherein the network host information is collected by at leastone network host monitor.
 26. The information storage medium of claim20, further comprising communicating the new network flow profile with aservice provider.
 27. The information storage medium of claim 19,wherein applying the policy to the network flow includes communicatingidentifying information of the network flow to a network traffic controldevice.
 28. The information storage medium of claim 27, wherein theidentifying information of the network flow is communicated with thenetwork traffic control device via a network management protocol. 29.The information storage medium of claim 19, wherein the network trafficinformation is collected by at least one network traffic monitor.